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after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 
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3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
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Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
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* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Election 

[01] Claims 2-8,10-13,18 are withdrawn from furtlier consideration pursuant to 37 CFR 1 .142(b), as 
being drawn to a nonelected species, tliere being no allowable generic or linking claim. Applicant 
timely traversed the restriction (election) requirement in the reply filed on 3/1/10. 

[02] Applicant traverses because the application is a national stage application and because the claims 
satisfy the requirement of unity of invention. Applicant cites MPEP § 1850(11), which states that 
"[u]nity of invention has to be considered in the first place only in relation to the independent claims 
in an international application and not the dependent claims." Consequently, "[i]f the independent 
claims avoid the prior art and satisfy the requirement of unity of invention, no problem of lack of 
unity arises in respect of any claims that depend on the independent claims." 

[03] However, the cited portions of MPEP § 1 850(11) are addressed to claims depending on 

"independent claims that avoid the prior art:' (emphasis added). Claim 1 does not avoid the prior art 
(see below). 

[04] MPEP § 1 850(11) further states that "If, however, an independent claim does not avoid the prior art, 
then the question whether there is still an inventive link between all the claims dependent on that 
claim needs to be carefully considered. If there is no link remaining, an objection of lack of unity a 
posteriori (that is, arising only after assessment of the prior art) may be raised. Similar 
considerations apply in the case of a genus/species or combination/subcombination situation." 

[05] The species identified in the election requirement (1/27/10) share the technical features recited in 
claim 1 . As will be shown below, the technical feature of claim 1 is not a "special technical feature" 
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as it does not make a contribution over the prior art. The requirement is still deemed proper and is 
therefore made FINAL. 

Specification 

[06] The disclosure is objected to because of the following infonnalities: 

[06a] Paragraph [01 32] refers to "the above-descried [sic] energy minimization problem". 
[06b] Starting with paragraph [01 00] on page 29 of the specification, the equations are 

misnumbered. Paragraph [0100] refers to "the following equation (12)". What follows is 
labeled as "Expression 10". There is no issue with the specification altematively referring 
to "equations" versus "expressions". However, the numbers do require correction. The 
following are mislabeled: 

• "Expression 10" should be changed to "Expression 12" on page 29. 

• 1 1 should be 1 3 , 1 2 should be 1 4 on page 31 . 

• 1 3 should be 1 5 on page 34. 

• 14 should be 16, 15 should be 17 on page 36. 

• 1 6 should be 1 8, 1 7 should be 1 9 on page 37. 
[06c] Appropriate con-ection is required. 

Claim Rejections - 35 USC § 112, Paragraph One 

[07] The following Is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of 
the manner and process of making and using it, in such full, clear, concise, 
and exact terms as to enable any person skilled in the art to which it 
pertains, or with which it is most nearly connected, to make and use the 
same and shall set forth the best mode contemplated by the inventor of 
carrying out his invention. 
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[08] Claim 1 6 is rejected under 35 U.S.C. 1 12, first paragraph, as failing to comply with the written 
description requirement. The claim(s) contains subject matter which was not described in the 
specification in such a way as to reasonably convey to one skilled in the relevant art that the 
inventor(s), at the time the application was filed, had possession of the claimed invention. 

[09] Claim 1 6 recites "an omnidirectional camera". Applicant elected an embodiment wherein the camera 
is "mounted on a tip of a probe". Figures 1 and 2 show a probe-type endoscope having a camera 
labeled "32". According to the specification ([0094]), this camera "32" is "an omnidirectional 
camera". However, Figures 1 and 2 simply do not support the assertion that the camera "32" is 
omnidirectional. 

[1 0] Figure 3 shows a detail of the "omnidirectional camera 32" comprising a "hyperboloidal mirror 42". 
But the detail of "32" can not be reconciled with the probe-type endoscope of Figures 1 and 2. 
According to the detail, the mirror "42" reflects light from the sides toward a sensor in the middle. 
According to Figures 1 and 2, the "omnidirectional camera 32" does not receive light from the sides 
but from the front. 

Claim Rejections - 35 U.S.C. 112, Paragrapli Two 

[11] The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing 
out and distinctly claiming the subject matter which the applicant regards as 
his invention. 

[12] Claims 9,14-16 are rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing 
to particularly point out and distinctly claim the subject matter which applicant regards as the 
invention. 
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[1 3] Claim 9 recites a panoramic image generation unit "operable to define predetermined energy." It is 
not clear what "predetennined energy" refers to. Moreover, it is not clear what exactly the panoramic 
image generation unit is capable of defining if that something is "predetermined". There is also an 
antecedent basis problem since later claims refer back to "the predetermined energy". 
[1 3a] For the purposes of applying the prior art, this portion of claim 9 will be interpreted as 
"operable to define pr e d e t e rm i n e d an.energy". The referential portion of claim 14 will be 
interpreted as "the prodotorm i nod energy". 

[14] Claims 14-1 5 incorporate claim 9, and are rejected on that basis. 

[1 5] Claim 1 6 recites "a panoramic image having a fixed visual angle with respect to a direction 
perpendicular to a traveling direction of said omnidirectional camera". It is not clear what a 
"panoramic image having a fixed visual angle" is. It is clear from the disclosure that the 
omnidirectional camera has a traveling direction, and it is understood that any number of lines could 
be imagined that are perpendicular to that direction. But it is not clear what "a fixed visual angle with 
respect to" this set of perpendicular lines would be. Nor is it clear what property of the panoramic 
image is affected by this "fixed visual angle". 

Claim Rejections - 35 USC § 103 

[16] The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections 

set forth in this Office action: 

A patent may not be obtained though the invention is not identically 
disclosed or described as set forth in section 1 02 of this title, if the 
differences between the subject matter sought to be patented and the prior 
art are such that the subject matter as a whole would have been obvious at 
the time the invention was made to a person having ordinary skill in the art 
to which said subject matter pertains. Patentability shall not be negatived by 
the manner in which the invention was made. 
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[17] Claims 1 ,9,14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Gombrich 
(6,081 ,740) in view of Heung-Yeung Shum and R. Szeliski, "Construction and Refinement of 
Panoramic Mosaics with Global and Local Alignment," Proc. Int'l Conf. Computer Vision, pp. 953- 
958, 1998. 

[18] Witli regard to claim 1 : 

[1 8a] Gombrich discloses an endoscope system ("endoscope 30") for tailing images of an inside 
of an object, comprising: 

• a camera ("imaging detectors 45" 4/1 7) operable to take images of the inside of the 
object in a living body, which is capable of motion; and 

• an image generation unit ("computing device 1 1 0" 5/1 3) operable to generate a 
panoramic image of the inside of the object by performing a video mosaicking process 
("joining multiple contiguous fields of view to produce a panoramic display" 5/22-23) 
intended for pasting images. 

[1 8b] Gombrich does not disclose a motion correction process, or an image modification process 
through energy minimization on the plurality of images obtained by said camera. 

[1 8c] Heung-Yeung (hereafter "HY") discloses a motion con-ection process ("motion estimate" 
p957), and an image modification process through energy minimization on the plurality of 
images obtained by said camera ("minimize the difference between screen coordinates of 
all overlapping pairs of images" p955), said processes being intended for estimating 
camera motion ("camera translation" p957), correcting previously definable motion in the 
living body and correcting previously indefinable internal defomiation in the living body 
("moving objects") p957. 
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[1 Sd] At the time of the invention, it would have been obvious to a person of ordinary skill in the 
art to provide motion con-ection and image modification through energy minimization on 
the images captured by Gombrich's device. A skilled artisan would be motivated to do so 
in order to generate a panoramic image that is free from misregistrations caused by 
camera translation or moving objects, as taught by HY. 
[1 9] With regard to claim 9: the image generation unit disclosed by Gombrich in view of HY comprises: 

[1 9a] a feature region cutout unit operable to cut out a plurality of feature regions ("use the patch 
centers as prospective "feature" points" p955) having a predetermined size ("e.g., 16x16 
pixels" p955) from each of the plurality of images obtained by said camera; and 

[1 9b] a panoramic image generation unit operable to define an energy based on the plurality of 
feature regions included in each of the plurality of images, associate the plurality of feature 
regions between the plurality of images such that the energy is minimized, and generate a 
panoramic image of the inside of the object based on the association result ("minimize the 
difference between the ray directions of con'esponding points using a rotational panoramic 
representation with unknown focal length" p955). 
[20] With regard to claim 14: the predetermined energy is determined based on a degree of deviation 

between a plurality of control points selected from a first image taken by said camera and a plurality 

of control points, in a second image taken by said camera, which respectively correspond to the 

plurality of control points selected from the first image ("con-esponding points between pairs of 

images are automatically obtained using patch-based alignment" p955). 

Additional Claim Rejections - 35 USC § 103 
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[21] Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Gombrich (6,081 ,740) in 
view of Heung-Yeung Shum, and in further view of Belson (2005/0165276). 

[22] As noted above, Gombrich in view of HY discloses a camera in a probe that is to be inserted into a 
digestive organ. 

[23] Gombrich in view of HY does not disclose that the camera is in the tip of the probe. 
[24] Belson discloses a "CCD or CMOS camera, positioned at the distal end 108 of the endoscope body 
102" ([0027]). 

[25] At the time of the invention, it would have been obvious to a person of ordinary skill in the art to 

provide a camera at the distal tip of the probe-type endoscope disclosed by Gombrich. A skilled 
artisan would be motivated to do so in order to obviate the image relay elements disclosed by 
Gombrich. 

Allowable Subject Matter 

[26] Claims 15-16would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112, 2nd 
paragraph, set forth in this Office action and to include all of the limitations of the base claim and 
any intervening claims. 

Conclusion 

[27] The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. 

[28] Akizuki (2007/0073103), Gilad (2006/0217593), Wallace (2006/0106283), Iddan (2004/0249247) 
disclose combining multiple ring-shaped images into a long cylindrical image. Glukhovsky 
(2003/0120130) discloses an algorithm that combines forward-looking images and backward- 
looking images based on pattern matching. Hale (2004/0210105) relies on accurate measurement 
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of the camera position and direction. None of these methods includes an "energy minimization" 
technique. Arai (2003/0191369) shows an omnidirectional endoscope. 

[29] Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to Philip R Smith whose telephone number is (571) 272 6087 and whose email address is 
philip.smith@uspto.gov. The examiner can normally be reached between 9:00am and 5:00pm. 

[30] If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Linda 
Dvorak can be reached on (571) 272 4764. 

[31 ] Information regarding the status of an application may be obtained from the Patent Application 

Information Retrieval (PAIR) system. Status information for published applications may be obtained 

from either Private PAIR or Public PAIR. Status information for unpublished applications is 

available through Private PAIR only. For more infonnation about the PAIR system, see http://pair- 

direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the 

Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

/Philip R Smith/ 
Examiner, Art Unit 3739 



